STORM-2355: Storm-HDFS: inotify support by tibkiss · Pull Request #1937 · apache/storm

tibkiss · 2017-02-12T10:57:49Z

No description provided.

Change-Id: I5e8a81738ff59afb8073a75c604bde2c1ef5b8d1

Change-Id: I0e2a8b0a15c090629cc5b220bb5891cdcabd7659

harshach · 2017-02-25T03:20:01Z

external/storm-hdfs/src/main/java/org/apache/storm/hdfs/common/HdfsDirectoryMonitor.java

+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.*;


avoid wild-card imports.

harshach · 2017-02-25T03:23:53Z

external/storm-hdfs/src/test/java/org/apache/storm/hdfs/spout/TestHdfsSpout.java

-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;


harshach · 2017-02-25T03:24:15Z

external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java

+        return null;
+      }
+      for (Path file : hdfsDirectoryMonitor) {
+        if (Path.getPathWithoutSchemeAndAuthority(file).toString().startsWith(lockDirPath.toString())) {


we can merge these two if blocks into single if block.

harshach · 2017-02-25T03:25:09Z

pom.xml

        <disruptor.version>3.3.2</disruptor.version>
        <jgrapht.version>0.9.0</jgrapht.version>
-        <guava.version>16.0.1</guava.version>
+        <guava.version>14.0.1</guava.version>


any reason to revert to a older version of guava

Hdfs dependency hell story: hadoop depends on 14.0.1 guava. If guava 16 is used & inotify calls are made the following exception occurs:

org.apache.hadoop.ipc.RemoteException:
com.google.common.collect.ComparisonChain.compare(ZZ)Lcom/google/common/collect/ComparisonChain;
at org.apache.hadoop.hdfs.server.namenode.JournalSet$1.compare(JournalSet.java:67)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$1.compare(JournalSet.java:62)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:324)
at java.util.TimSort.sort(TimSort.java:189)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at java.util.Collections.sort(Collections.java:217)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.chainAndMakeRedundantStreams(JournalSet.java:333)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:284)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1508)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1532)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getEditsFromTxid(NameNodeRpcServer.java:1997)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolServerSideTranslatorPB.java:1496)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)

at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy37.getEditsFromTxid(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEditsFromTxid(ClientNamenodeProtocolTranslatorPB.java:1511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy38.getEditsFromTxid(Unknown Source) at org.apache.hadoop.hdfs.DFSInotifyEventInputStream.poll(DFSInotifyEventInputStream.java:111) at org.apache.storm.hdfs.common.HdfsUtils.listNewFiles(HdfsUtils.java:94) at org.apache.storm.hdfs.spout.TestHdfsSpout.testListNewFiles(TestHdfsSpout.java:549)

tibkiss · 2017-02-25T05:16:32Z

Thanks for the comments. I'll be OoO next week, will address them beginning of March.

roshannaik · 2017-02-27T08:00:57Z

@tibkiss on the JIRA I had a few questions to get clarity around this.
That would help get a perspective to evaluate this. Could you please respond to them ?

tibkiss · 2017-03-06T13:31:43Z

@roshannaik : I've replied to your questions on the JIRA by extending the issue description. Please check it out and let me know if you have further questions. Thanks!

Change-Id: I932f16f267459b2447ca611d10304cbea7d879d4

tibkiss · 2017-03-06T13:38:40Z

@harshach : Updated this PR based on your review.

tibkiss · 2017-03-08T20:01:43Z

bump

roshannaik · 2017-03-14T02:07:24Z

@tibkiss thanks for adding the details on the jira. that is useful. i have started looking into it.

We are closing stale Pull Requests to make the list more manageable. Please re-open any Pull Request that has been closed in error. Closes apache#608 Closes apache#639 Closes apache#640 Closes apache#648 Closes apache#662 Closes apache#668 Closes apache#692 Closes apache#705 Closes apache#724 Closes apache#728 Closes apache#730 Closes apache#753 Closes apache#803 Closes apache#854 Closes apache#922 Closes apache#986 Closes apache#992 Closes apache#1019 Closes apache#1040 Closes apache#1041 Closes apache#1043 Closes apache#1046 Closes apache#1051 Closes apache#1078 Closes apache#1146 Closes apache#1164 Closes apache#1165 Closes apache#1178 Closes apache#1213 Closes apache#1225 Closes apache#1258 Closes apache#1259 Closes apache#1268 Closes apache#1272 Closes apache#1277 Closes apache#1278 Closes apache#1288 Closes apache#1296 Closes apache#1328 Closes apache#1342 Closes apache#1353 Closes apache#1370 Closes apache#1376 Closes apache#1391 Closes apache#1395 Closes apache#1399 Closes apache#1406 Closes apache#1410 Closes apache#1422 Closes apache#1427 Closes apache#1443 Closes apache#1462 Closes apache#1468 Closes apache#1483 Closes apache#1506 Closes apache#1509 Closes apache#1515 Closes apache#1520 Closes apache#1521 Closes apache#1525 Closes apache#1527 Closes apache#1544 Closes apache#1550 Closes apache#1566 Closes apache#1569 Closes apache#1570 Closes apache#1575 Closes apache#1580 Closes apache#1584 Closes apache#1591 Closes apache#1600 Closes apache#1611 Closes apache#1613 Closes apache#1639 Closes apache#1703 Closes apache#1711 Closes apache#1719 Closes apache#1737 Closes apache#1760 Closes apache#1767 Closes apache#1768 Closes apache#1785 Closes apache#1799 Closes apache#1822 Closes apache#1824 Closes apache#1844 Closes apache#1874 Closes apache#1918 Closes apache#1928 Closes apache#1937 Closes apache#1942 Closes apache#1951 Closes apache#1957 Closes apache#1963 Closes apache#1964 Closes apache#1965 Closes apache#1967 Closes apache#1968 Closes apache#1971 Closes apache#1985 Closes apache#1986 Closes apache#1998 Closes apache#2031 Closes apache#2032 Closes apache#2071 Closes apache#2076 Closes apache#2108 Closes apache#2119 Closes apache#2128 Closes apache#2142 Closes apache#2174 Closes apache#2206 Closes apache#2297 Closes apache#2322 Closes apache#2332 Closes apache#2341 Closes apache#2377 Closes apache#2414 Closes apache#2469

tibkiss added 2 commits February 12, 2017 11:49

STORM-2355: Storm-HDFS: inotify support (initial version)

dc9aec2

Change-Id: I5e8a81738ff59afb8073a75c604bde2c1ef5b8d1

Handle leftover files in HdfsSpout's input directory

3a6ef55

Change-Id: I0e2a8b0a15c090629cc5b220bb5891cdcabd7659

harshach reviewed Feb 25, 2017

View reviewed changes

Addressing reviewer's comments about coding style

b1a13f4

Change-Id: I932f16f267459b2447ca611d10304cbea7d879d4

d2r mentioned this pull request Oct 16, 2018

STORM-3250: Closes Pull Requests unchanged in 2018 #2880

Merged

asfgit closed this in #2880 Oct 22, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

STORM-2355: Storm-HDFS: inotify support#1937

STORM-2355: Storm-HDFS: inotify support#1937
tibkiss wants to merge 3 commits intoapache:1.x-branchfrom
tibkiss:feature/inotify_in_hdfsspout-1.x-branch

tibkiss commented Feb 12, 2017

Uh oh!

harshach Feb 25, 2017

Uh oh!

harshach Feb 25, 2017

Uh oh!

harshach Feb 25, 2017

Uh oh!

harshach Feb 25, 2017

Uh oh!

tibkiss Feb 25, 2017 •

edited

Loading

Uh oh!

tibkiss commented Feb 25, 2017

Uh oh!

roshannaik commented Feb 27, 2017

Uh oh!

tibkiss commented Mar 6, 2017 •

edited

Loading

Uh oh!

tibkiss commented Mar 6, 2017

Uh oh!

tibkiss commented Mar 8, 2017

Uh oh!

roshannaik commented Mar 14, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tibkiss commented Feb 12, 2017

Uh oh!

harshach Feb 25, 2017

Choose a reason for hiding this comment

Uh oh!

harshach Feb 25, 2017

Choose a reason for hiding this comment

Uh oh!

harshach Feb 25, 2017

Choose a reason for hiding this comment

Uh oh!

harshach Feb 25, 2017

Choose a reason for hiding this comment

Uh oh!

tibkiss Feb 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tibkiss commented Feb 25, 2017

Uh oh!

roshannaik commented Feb 27, 2017

Uh oh!

tibkiss commented Mar 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tibkiss commented Mar 6, 2017

Uh oh!

tibkiss commented Mar 8, 2017

Uh oh!

roshannaik commented Mar 14, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tibkiss Feb 25, 2017 •

edited

Loading

tibkiss commented Mar 6, 2017 •

edited

Loading